The Dimensions and Complexities of Audio-Visual Association

This paper draws on literature from psychology, neuroscience, linguistics, and philosophy to portray a cross-modal conception of auditory and visual phenomena focusing on the similarities in form, structure, and function and exploring the links to perception, conception, and language. Through an extensive literature review, we identify issues related to audio-visual association and explore divergences and convergences between the two modalities. The aim of this paper is to examine how recent research findings from brain science could inform the theoretical and methodological approaches used for studying similarity of auditory and visual percepts in the context of designing multimodal interaction.